Systems and methods for managing disaster recovery in a storage system

ABSTRACT

Systems and methods for providing for efficient switchover for a client in a storage network between the use of one or more a primary storage resources to one or more disaster recovery (DR) resources are provided herein. Embodiments may implement synchronization between such resources on a data plane and a control plane to allow for a transition between resources to be implemented in a manner that is minimally disruptive to a client. Moreover, embodiments may provide for processing resources which allow for switching a client between a primary storage resource to a secondary storage resource with minimal administrative interaction.

TECHNICAL FIELD

The present invention relates generally to management of data withinfile systems and, more particularly, management of data which ismirrored in multiple locations in a storage network.

BACKGROUND OF THE INVENTION

The creation and storage of digitized data has proliferated in recentyears. Accordingly, techniques and mechanisms that facilitate efficientand cost effective storage of large amounts of digital data are commontoday. For example, a cluster network environment of nodes may beimplemented as a data storage system to facilitate the creation,storage, retrieval, and/or processing of digital data. Such a datastorage system may be implemented using a variety of storagearchitectures, such as a network-attached storage (NAS) environment, astorage area network (SAN), a direct-attached storage environment, andcombinations thereof. The foregoing data storage systems may compriseone or more data storage devices configured to store digital data withindata volumes.

Digital data stored by data storage systems may be frequently migratedand replicated within the data storage system and/or between datastorage systems during normal operation. For example, data storagesystems may include disaster recovery (DR) resources that function as abackup for a primary storage resource. Other networks may provide forhigh availability (HA) resources that are configured to assist a primarystorage resource when desired, e.g. during high traffic periods, toprovide for faster data transfers. Such DR and HA systems obtainreplicated information of data residing on a primary storage resource.

In order to replicate information from a primary storage resource to asecondary storage resource, previous systems may utilize write cachethat compiles write data from a client which is then stored to a morepermanent storage location according to a pre-determined schema. Forexample, the write cache may be part of a DR node and function tocompile data for the DR node until the cache resources are near full (oruntil another pre-determined condition is met), at which point the dataresiding on the write cache is transferred to DR storage. The point whenwrite cache is cleared and its contents sent to storage may be referredto as a consistency point.

It is notable, however, that while data is replicated to DR resources,presently the transition between the actual use of a primary storageresource and a DR resource is time consuming and disruptive to a client.In many instances, the mirrored data is mirrored with a significantdelay in time. As such, not all of the stored data may be available to aclient in the event that the primary node ceases functioning. Further,in making the transition between resources, various controls andconfiguration information must be established in order to properlyinsure communications between a client and the correct transitionedstorage volumes. Establishing such controls is time consuming and canbecome administratively complicated. In fact, establishing communicationconfigurations is often accomplished on a manual basis where anadministrator must manually select and assign resources for a client.

It is further noted that the above-mentioned issues found intransitioning between a primary resource and a DR resource may also beapparent when implementing switchback procedures. For example, a primaryresource may be brought back online and resynchronize data with the DRnode, but the requisite control data and commands to administer theprimary storage resource will still need to be reestablished using thesame methods.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention, reference isnow made to the following descriptions taken in conjunction with theaccompanying drawings, in which:

FIG. 1 provides a block diagram of a storage system in accordance withan embodiment of the present invention;

FIG. 2 illustrates a DR/HA storage network in accordance with anembodiment of the present application;

FIG. 3 illustrates a state diagram corresponding to various states ofthe DRC in accordance with an embodiment of the present application;

FIG. 4 illustrates a method for administering DR functionality in astorage network in accordance with an embodiment of the presentapplication; and

FIG. 5 illustrates a method for administering DR functionality within astorage network in accordance with an embodiment of the presentapplication.

DETAILED DESCRIPTION OF THE INVENTION

The present application provides for systems and methods which implementefficient switchover for a client in a storage network between the useof one or more a primary storage resources to one or more secondaryresources, such as DR resources. Embodiments may implementsynchronization between such resources on a data plane and a controlplane to allow for a later transition between resources to beimplemented in a manner that is minimally disruptive to a client.

On the data plane, embodiments may utilize an asynchronous resource(e.g. a RAID system and the like) and a near synchronous write cacheresource. The write cache may receive data from a client device andsynchronize the data with the asynchronous resource at various points intime to create consistency points. On the control plane, embodiments mayprovide for one or more virtual servers that include configurationinformation for the storage network to be replicated to secondaryresources and remain dormant on the secondary resources while theprimary storage resources are functional. Upon receiving a switchovercommand, one or more dormant virtual servers may be activated in amanner that allows an efficient transition between resources for aclient.

Embodiments may also provide for processing resources at various levelsin a storage network, e.g. at a management level, a cluster level and anode level, which may coordinate in a manner which simplifies switchoverprocedures for an administrator commanding a switchover. For example,one or more nodes may include processing capabilities which receive aswitchover command from the storage system (e.g. from an administrator)and translates that command into multiple tasks that must be implementedin order to perform the switchover task. Such processing capabilitiesmay also function to monitor these tasks within various portions of anode or cluster and provide feedback where appropriate to allow othertasks to complete.

The foregoing has outlined rather broadly the features and technicaladvantages of the present invention in order that the detaileddescription of the invention that follows may be better understood.Additional features and advantages of the invention will be describedhereinafter which form the subject of the claims of the invention. Itshould be appreciated by those skilled in the art that the conceptionand specific embodiment disclosed may be readily utilized as a basis formodifying or designing other structures for carrying out the samepurposes of the present invention. It should also be realized by thoseskilled in the art that such equivalent constructions do not depart fromthe spirit and scope of the invention as set forth in the appendedclaims. The novel features which are believed to be characteristic ofthe invention, both as to its organization and method of operation,together with further objects and advantages will be better understoodfrom the following description when considered in connection with theaccompanying figures. It is to be expressly understood, however, thateach of the figures is provided for the purpose of illustration anddescription only and is not intended as a definition of the limits ofthe present invention.

FIG. 1 provides a block diagram of storage system 100 in accordance withan embodiment of the present invention. System 100 includes a storagecluster having multiple nodes 110 and 120 which are adapted tocommunicate with each other and any additional node of the cluster.Nodes 110 and 120 are configured to provide access to data stored on aset of storage devices (shown as storage devices 114 and 124)constituting storage of system 100. Storage services may be provided bysuch nodes implementing various functional components that cooperate toprovide a distributed storage system architecture of system 100.Additionally, one or more storage devices, such as storage array 114,may act as a central repository for storage system 100. It isappreciated that embodiments may have any number of edge nodes such asmultiple nodes 110 and/or 120. Further, multiple storage arrays 114 maybe provided at the multiple nodes 110 and/or 120 which provide resourcesfor mirroring a primary storage data set.

Illustratively, nodes (e.g. network-connected devices 110 and 120) maybe organized as one or more network elements (N-modules 112 and 122)and/or storage elements (D-modules 113 and 123) and a management element(M-host 111 and 121). N-modules may include functionality to enablenodes to connect to one or more clients (e.g. network-connected device130) over computer network 101, while D-modules may connect to storagedevices (e.g. as may implement a storage array). M-hosts may providecluster communication services between nodes for generating informationsharing operations and for presenting a distributed file system imagefor system 100. Functionality for enabling each node of a cluster toreceive name and object data, receive data to be cached, and tocommunicate with any other node of the cluster may be provided byM-hosts adapted according to embodiments of the invention.

It should be appreciated that network 101 may comprise various forms,and even separate portions, of network infrastructure. For example,network-connected devices 110 and 120 may be interconnected by clusterswitching fabric 103 while network-connected devices 110 and 120 may beinterconnected to network-connected device 130 by a more general datanetwork 102 (e.g. the Internet, a LAN, a WAN, etc.).

It should also be noted that while there is shown an equal number of N-and D-modules constituting illustrated embodiments of nodes, there maybe a different number and/or type of functional components embodyingnodes in accordance with various embodiments of the present invention.For example, there may be multiple N-modules and/or D-modulesinterconnected in system 100 that do not reflect a one-to-onecorrespondence between the modules of network-connected devices 110 and120. Accordingly, the description of network-connected devices 110 and120 comprising one N- and one D-module should be taken as illustrativeonly and it will be understood that the novel technique is not limitedto the illustrative embodiment discussed herein.

Network-connected device 130 may be a general-purpose computerconfigured to interact with network-connected devices 110 and 120 inaccordance with a client/server model of information delivery. To thatend, network-connected device 130 may request the services ofnetwork-connected devices 110 and 120 by submitting a read or writerequest to the cluster node comprising the network-connected device. Inresponse to the request, the node may return the results of therequested services by exchanging information packets over network 101.Network-connected device 130 may submit access requests by issuingpackets using application-layer access protocols, such as the CommonInternet File System (CIFS) protocol, Network File System (NFS)protocol, Small Computer Systems Interface (SCSI) protocol encapsulatedover TCP (iSCSI), and SCSI encapsulated over Fibre Channel (FCP) forinstance.

System 100 may further include a management console (shown here asmanagement console 150) for providing management services for theoverall cluster. Management console 150 may, for instance, communicatewith network-connected devices 110 and 120 across network 101 to requestoperations to be performed at the cluster nodes comprised of thenetwork-connected devices, and to request information (e.g. nodeconfigurations, operating metrics) from or provide information to thenodes. In addition, management console 150 may be configured to receiveinputs from and provide outputs to a user of system 100 (e.g. storageadministrator) thereby operating as a centralized management interfacebetween the administrator and system 100. In the illustrativeembodiment, management console 150 may be networked to network-connecteddevices 110-130, although other embodiments of the present invention mayimplement management console 150 as a functional component of a node orany other processing system connected to or constituting system 100.

Management console 150 may also include processing capabilities and codewhich is configured to control system 100 in order to allow formanagement of switchover between resources on storage system 100. Forexample, system 100 may include a primary storage node for storing dataand multiple secondary nodes which are configured act as DR resourcesfor the primary node. In such an example system, management console 150may establish/control which nodes are utilized for DR resources, controlmessages between a client and the selected nodes, monitor the nodes forfailures, control switchover between nodes in the event of a failure andcontrol resynchronization of nodes coming back online after failures.These processing functions may also be shared between, or may beimplemented in cooperation with one or more M-hosts 111 and 121 withinnodes 110 and 120.

In a distributed architecture, network-connected device 130 may submitan access request to a node for data stored at a remote node. As anexample, an access request from network-connected device 130 may be sentto network-connected device 120 which may target a storage object (e.g.volume) on network-connected device 110 in storage 114. This accessrequest may be directed through network-connected device 120 due to itsproximity (e.g. it is closer to the edge than a device such asnetwork-connected device 110) or ability to communicate more efficientlywith device 130. To accelerate servicing of the access request andoptimize cluster performance, network-connected device 120 may cache therequested volume in local memory or in storage 124. For instance, duringinitialization of network-connected device 120 as a cluster node,network-connected device 120 may request all or a portion of the volumefrom network-connected device 110 for storage at network-connecteddevice 120 prior to an actual request by network-connected device 130for such data.

As can be appreciated from the foregoing, in order to operate as acluster (e.g. the aforementioned data storage system), network-connecteddevices 110-130 may communicate with each other. Such communication mayinclude various forms of communication (e.g. point-to-point or unicastcommunication, multicast communication, etc.). Accordingly, toeffectively cooperate to provide desired operation as a logical entity,each node of a cluster is provided with the capability to communicatewith any and all other nodes of the cluster according to embodiments ofthe invention.

FIG. 2 illustrates a DR/HA storage network 200 in accordance with anembodiment of the present application. It is noted that while much ofthe discussion herein of embodiments and functionality of the inventionis made in the context of being implemented in a DR/HA environment,embodiments may be implemented within many types of networks (such asstorage system 100). Further, various nodes and devices of storagenetwork 200 may be implemented in the manner as discussed above withrespect to the devices and functionality of storage system 100.

DR/HA storage network 200 includes a first cluster 210 having primarynode 211 and HA node 212, and a second cluster 220 which is remotelylocated with respect to first cluster 210 and has DR node 221 and DR/HAnode 222. A client 230 may write data which is directed to primary node211 of first cluster 210 via communication network 240. Additionally,management server 250 comprises processing circuitry which is configuredto provide for control of the information flow across DR/HA storagenetwork 200.

It is noted that various portions of DR/HA storage network 200 (e.g. thenodes, communications, clients, management console/server etc.) may beimplemented as discussed above with respect to storage system 100. It isfurther noted that first cluster 210 and second cluster 220 may includeadditional nodes, however, the presently illustrated embodiment has beensimplified for the sake of describing the novel concepts provided forherein. Additionally, the designation of primary node 211, HA node 212and DR node 221 are set forth with respect to client 230. In someembodiments, a different client may utilize DR node 221 as a primarynode, DR/HA node 222 as an HA node, and primary node 211 would be the DRnode with respect to the different client. Such clients may existsimultaneously in DR/HA storage network 200 in accordance withembodiments described herein. Further, management server 250 may beimplemented as a stand-alone processing server, may function inconjunction with processing capabilities of one or more nodes, or may beimplemented entirely within the processing capabilities of one or morenodes.

In one use example, management server 250 may establish a primary nodefor storing information, such as primary node 211, for client device230. Management server 250 may also establish one or more secondarynodes which are configured to mirror the information written to primarynode 211. Determining the primary and secondary nodes may beaccomplished manually or automatically. For example, the designation ofa primary node and secondary nodes may be implemented by anadministrator with access to management server 250. Additionally,automated determinations may be implemented where a determination isbased on one or more system properties (e.g. capacity of particularnodes, communication capabilities and/or bandwidth between nodes, etc.).In the illustrated DR/HA storage network 200, client 230 may be assignedHA node 212 and DR node 221 as mirroring nodes. In this manner, client230 may initiate a write command which causes data to be written ormodified in primary node 211, whereupon management server 250 causes thedata to also be written within DR node 221. If HA functionality isdesired, management server 250 may also cause the data to be writtenwithin HA node 212.

Management server 250 may be configured to manage the node partitionsused by a client and retain information regarding addressing/extensionsof these partitions and any other information which is helpful tocontrol data flow and monitor the functions of devices communicatingwithin DR/HA network 200. For example, management server 250 may beconfigured to monitor the network devices for various states such asstates which reflect whether: the primary storage node 211 is online,the HA node 212 is online and synchronized with primary node 211, andwhether the DR node is online and synchronized with the primary storagenode 211 and/or HA node 212. States which reflect whether mirror deviceshave lost communication, or whether switchovers have occurred (e.g. aswitchover from a primary node to a DR node), etc., may also bemonitored. Such monitoring may be implemented in any manner whichprovides relevant data to management server 250. For example, variousportions of DR/HA network 200 may be configured to provide a periodicsignal (e.g. a heartbeat signal) to one or more devices in the network.Management server 250 may monitor such signals to determine whether aparticular device is functioning properly. Additionally, in someembodiments, management server 250 may periodically ping one or moredevices in DR/HA network 200 and monitor the response to determinewhether a particular device is functioning properly. Further, in someinstances, a device which is experiencing abnormal operations may alsobe configured to provide error messages to management server 250. Whenmanagement server 250 discovers a problem (e.g. receives an errormessage, fails to receive a heartbeat signal or a response to a testsignal, etc.) within DR/HA storage network 200, management server 250may initiate failure procedures.

Each node of FIG. 2 is illustrated as including a write cache (WC)portion 213 and a storage portion 214 which are designated for client230. It is noted that write cache 213 may temporarily contain datawritten by client 230 and may contain metadata which assists in thefunctioning of the overall storage network 200. Storage portion 214contains storage data on a more permanent basis. In some embodimentswrite cache portion 213 may store data on a device which is separatefrom data stored on storage portion 214, while in other embodiments,such storage may be implemented on the same device. Further, asillustrated, storage portion 214 is part of an overall node, however,such storage may be implemented within its own array that is incommunication with a node (such as illustrated with respect to storage114 and 124 of FIG. 1).

Each node of FIG. 2 also includes a virtual server (VS) 215 and 216. Itis noted, however, that a virtual server may also provide for thevirtual aggregation of resources from different nodes that are accessedby clients via a protocol such as iSCSI or NFS multiple nodes in acluster. A virtual server may maintain information and control data flowbetween a client and storage resources. For example, a client may nothave awareness over where data associated with that client resides in acluster, which nodes or disk drives are assigned to the client forfuture storage, and the like. Embodiments may synchronize these virtualservers to facilitate fast switchover of resources in the event that aprimary node is taken offline.

In one embodiment, client 230 may be assigned to an active virtualserver VS1 215 whereupon configuration information, e.g. informationregarding what portion of the storage resources are being used by theclient and information from other subsystems that handle configuration(such as protocol specific control information, etc.), are known by VS1215. VS1 215 may be replicated on DR node 221 at a dormant virtualserver VS1 215 whereupon configuration information regarding whatportion of the DR storage resources are, being used by client 230 may bemaintained.

Accordingly, for DR resources embodiments may replicate datacorresponding to a client on both a data plane and a control plane. Thedata plane may include any data structures which facilitate the storageand reading of payload data from a client. The data plane comprises bothan asynchronous component(s) and a synchronous component(s) (or nearsynchronous due to transmission delay and the like). Specifically, writecache 213 on node 211 may be synchronously replicated with write cache213 on DR node 221. Storage portion 214 of node 211 may be replicated tostorage 214 on DR node 211 in an asynchronous manner utilizingconsistency points and write cache 213. The control plane may includecomponents and processing resources which implement or control where orhow data is stored on the data plane (e.g. which resources of storageportion 214 are utilized and the like). On the control plane, virtualservers, such as VS1 215 may be replicated at a separate VS1 215. Inthis case, when a primary node becomes nonfunctional, VS1 215 may becomedormant and the previously dormant VS1 215 may become the active virtualserver.

With both the data plane and control plane information, embodiments mayimplement switching between nodes, e.g. in the event of a DR situation,in a manner that provides minimal interruption to a client. Further, insome instances, a switchover may be implemented in a manner that istransparent to the client. For example, configuration information may bereplicated among virtual servers such that a client will continue torecognize that communication paths are available for storing data, evenif such paths are changing, in order to prevent an error whichinterrupts the client's functionality.

Additionally, embodiments may implement a switchback routine in asimilar manner. For example, in the event that primary node 211 losescommunication within DR/HA storage network 200, storage network 200 maycause client 230 to utilize DR resources after transitioning asdescribed above. When primary node 211 comes back online, storagenetwork 200 may cause the appropriate information from DR node 221 to bereplicated back to primary node 211 in order to allow for a switchbackroutine.

It is noted that write cache 213, storage 214 and virtual servers 215may be mirrored to other nodes under control of management server 250.Alternatively, management server 250 may provide primary node 211 withthe information on where to send mirror data and primary node 211 maysend multiple mirroring data signals to other portions of network 200,e.g. to HA node 212 and DR node 221. Further control may be implementedby a combination of management server 250 and processing resourceswithin a node (e.g. by M-Host 121).

In some embodiments, a storage network such as networks 100 and 200 maybe further configured such that an administrator may cause theswitchover from a primary node to a secondary node (such as a DR node)with an explicit command, as opposed to prior art methods which requiremultiple steps, manual configurations, and the like. For example, anM-Host resource within a node such as nodes 110, 120, 211 and 221, maybe configured with a Disaster Recovery Coordinator (DRC) which isconfigured to convert a user command from an administrator into multipletasks for the storage network to implement. Specifically, the DRC mayidentify the nodes participating in a DR switchover operation, relegatethe appropriate task(s) to each node for the operation, wait for eachnode to complete the task, and, finally, notify the administrator of thestatus of the operation across the cluster.

A DRC may be responsible for mapping the remote nodes that need to beswitched-over and identify which nodes will participate in a switchoperation (and/or a switch back operation). This may be implemented withor without administrator assistance. The DRC may also act as a statemachine and monitor/manage the states of a transition between the use ofstorage resources.

FIG. 3 illustrates a state diagram 300 corresponding to various statesof the DRC in accordance with an embodiment of the present application.During normal operations of the storage network, the state of the DRCwill be at normal 301. State 302 reflects that a switchover is inprogress, such as a switchover of a client from a primary node to a DRnode. Such a switchover may be automatically caused as a result of anode going offline or may be started as a result of an administratorcommanding the switchover. In the event of a successful switchover (e.g.all nodes have been switched over correctly) the state reflects that theswitchover is completed at 303. If at least one node has switched over,but there are remaining nodes that have not successfully switched over,a partial switchover state is reflected at 304.

When the switchover has completed (or during the switchover), DRC maybegin healing the aggregates (assuming there is a problem with theprimary nodes) at 305. State 306 reflects that the healing is partiallycompleted with respect to the nodes and the healing continues until itis completed at state 307. At this point, healing of the root disks ofthe source cluster mode may begin at 308. Healing the roots causes theswitched over root aggregates to be switched back to the source clusterso that the source cluster nodes can boot up before switchback occurs.While root aggregates are not healed on all nodes (at 309), switchbackcommands will be rejected until the healing of the roots is completed at310. At that point, a switchback command may be implemented to cause theswitchback to begin at 311. The switchback will reflect as partiallycomplete at 312 until it is successful on all nodes and the statereturns to normal 301.

The DRC may also be in communication with a DR switch over monitor(DR-SOM) and DR disk ownership manager (DR-DOM) processingresources/modules residing in a node. DR-SOM may be implemented as areactive system that collects data from outside events and outputs datato change external environments. Specifically, DR-SOM mayprocess/interpret external commands and cause variousswitchover/switchback operations to occur on the node level. Forexample, during a switchover operation, DR-SOM coordinates theinteractions between sub-systems in a specified order. Each sub-systemperforms a specific task to complete the switchover or switchback.Instead of requiring an administrator to run a command to perform asub-system specific task, DR-SOM directly invokes the sub-system tocarry out that task. DR-DOM is an example of such sub-systems. DR-DOMmay comprise the sub-system concerned with disk ownership changes thatoccur during switchover or switchback. Additionally, DR-DOM may functionto receive commands from DR-SOM during a switchover process to changedisk ownership from, e.g., a disaster stricken site to a surviving site.During switchback, DR-SOM may invoke DR-DOM to revert disk ownershipback to the source site.

Accordingly, embodiments may provide for single command switchoverfunctionality where an administrator requests a switchover andprocessing resources on the storage network, e.g. the DRC, DR-SOM, andDR-DOM, and implement the coordination of each of the subsystemsrequired to change configurations during a switchover between clustersand nodes. It is further noted that these resources may functionsimilarly to provide a single command switch back operation.

In view of exemplary systems shown and described herein, methodologiesthat may be implemented in accordance with the disclosed subject matterwill be better appreciated with reference to various functional blockdiagrams. While, for purposes of simplicity of explanation,methodologies are shown and described as a series of acts/blocks, it isto be understood and appreciated that the claimed subject matter is notlimited by the number or order of blocks, as some blocks may occur indifferent orders and/or at substantially the same time with other blocksfrom what is depicted and described herein. Moreover, not allillustrated blocks may be required to implement methodologies describedherein. It is to be appreciated that functionality associated withblocks may be implemented by software, hardware, a combination thereofor any other suitable means (e.g. device, system, process, orcomponent). Additionally, it should be further appreciated thatmethodologies disclosed throughout this specification are capable ofbeing stored on an article of manufacture to facilitate transporting andtransferring such methodologies to various devices. Those skilled in theart will understand and appreciate that a methodology couldalternatively be represented as a series of interrelated states orevents, such as in a state diagram.

FIG. 4 illustrates a method 400 for administering DR functionality in astorage network in accordance with an embodiment of the presentapplication. It is noted that embodiments of method 400 may beimplemented within the storage networks described above with respect toFIGS. 1-3. Specifically, method 400 may include, at block 401,establishing at least one primary storage node and at least onesecondary storage node for a client.

At block 402, method 400 may include replicating a data plane of theprimary node(s) to the secondary node(s). Replicating the data plane mayinclude replicating both an asynchronous resource such as a drive in anode and/or storage array, and a synchronous resource such as a cachebuffer. Additionally, at block 403, a configuration plane for theprimary node(s) is replicated for use by the secondary node(s).Replication of the configuration plane may include replicatingconfiguration information relating to the location/storage of data froma client one or more storage resources. For example, replication ofconfiguration information may include replicating one or more virtualservers which function at a cluster level and administer to one or morenodes. Further, such replication may cause the virtual server on asecondary node to be dormant until activated at the point of aswitchover.

After the nodes are established and replication is taking place, method400 may include, at 404, monitoring for failure of primary storage node.In the event that no failure is present, the storage system continues tomonitor for failures at block 404. Upon recognizing a failure of theprimary node, method 400 may include at 405, switching the client fromhaving primary access with the primary storage node to having primaryaccess with the secondary storage node. Such a switch may be madeautomatically, or in response to a manual command provided by anadministrator. In some embodiments, the storage system may be configuredsuch that in response to a switch command, processing devices in thesystem may implement the switch without additional configurationinstructions from an administrator.

FIG. 5 illustrates a method 500 for administering DR functionalitywithin a storage network in accordance with an embodiment of the presentapplication. More specifically, method 500 provides an example flow forconducting a switchback operation. It is noted that as with the abovemethod, embodiments of method 500 may be implemented within the storagenetworks described above with respect to FIGS. 1-2. Method 500 begins atblock 501 with administering to a client device and providing storageresources at least one secondary storage node. At decision block 502, adetermination of whether a switchback process is desired is made. In theevent that a switchback is not wanted, method 500 continues functioningwith the at least one storage node at 501.

If a switchback operation is in order, at decision block 503 method 500determines whether the switch is switching back to one or more originalprimary storage nodes or switching to utilize different nodes as aprimary storage resource (e.g. different nodes within the originalcluster, or a different cluster). If switching to the original nodes,method 500 may include, at 504, healing the primary storage node(s).Once the primary storage nodes are healed, the data and configurationplanes are resynchronized at block 505 and the switchback may becompleted at 506 with the client switching back to utilizing nodes onthe primary storage resource for a primary storage node. Such aswitchback may be implemented in a similar manner as the originalswitchover.

In the event that the switchback will be directed to different nodesthan the original primary storage nodes, method 500 may includesynchronizing data and configuration planes with the new nodes at 507and switching the primary use from the secondary storage node to thenewly synchronized nodes at 508.

The functional blocks and modules of the above FIGS. 1-5 may compriseprocessors, electronics devices, hardware devices, electronicscomponents, logical circuits, memories, software codes, firmware codes,etc., or any combination thereof.

Those of skill would further appreciate that the various illustrativelogical blocks, modules, circuits, and algorithm steps described inconnection with the disclosure herein may be implemented as electronichardware, computer software stored on a computing device and executed byone or more processing devices, or combinations of both. To clearlyillustrate this interchangeability of hardware and software, variousillustrative components, blocks, modules, circuits, and steps have beendescribed above generally in terms of their functionality. Whether suchfunctionality is implemented as hardware or software depends upon theparticular application and design constraints imposed on the overallsystem. Skilled artisans may implement the described functionality invarying ways for each particular application, but such implementationdecisions should not be interpreted as causing a departure from thescope of the present disclosure.

It is also appreciated that the systems and method described herein areable to be scaled for larger storage network systems. For example, acluster may include hundreds of nodes, multiple virtual servers whichservice multiple clients and the like. Such modifications may functionaccording to the principles described herein.

Although the present invention and its advantages have been described indetail, it should be understood that various changes, substitutions andalterations can be made herein without departing from the spirit andscope of the invention as defined by the appended claims. Moreover, thescope of the present application is not intended to be limited to theparticular embodiments of the process, machine, manufacture, compositionof matter, means, methods and steps described in the specification. Asone of ordinary skill in the art will readily appreciate from thedisclosure of the present invention, processes, machines, manufacture,compositions of matter, means, methods, or steps, presently existing orlater to be developed that perform substantially the same function orachieve substantially the same result as the corresponding embodimentsdescribed herein may be utilized according to the present invention.Accordingly, the appended claims are intended to include within theirscope such processes, machines, manufacture, compositions of matter,means, methods, or steps.

What is claimed is:
 1. A method for administering disaster recoveryresources in a storage network, said method comprising: replicating, bya storage management computing device, a data plane and a configurationplane corresponding to at least one primary node to at least onesecondary storage node for a client device in communication with thestorage network; monitoring, by the storage management computing device,the at least one primary node for a failure in communication within thestorage network; switching, by the storage management computing device,primary access for the client device from the at least one primary nodeto the at least one secondary storage node in the event that a failurein the at least one primary node is recognized, the switching furthercomprising healing the at least one failed primary node and switchingprimary access for the client device back from the at least onesecondary storage node to the healed at least one primary node; andproviding, by the storage management computing device, a switchingstatus notification indicating progress of the switching primary accessfor the client device from the at least one primary node to the at leastone secondary storage node and a healing status notification indicatingprogress of healing the at least one failed primary node.
 2. The methodof claim 1 wherein replicating a data plane comprises replicating, bythe storage management computing device, an asynchronous resource and asynchronous resource configured to store data from the client device. 3.The method of claim 1 wherein replicating a configuration planecomprises replicating a virtual server.
 4. The method of claim 3 whereinthe replicated virtual server is configured to be dormant on the atleast one secondary storage node while the at least one primary node isfunctioning.
 5. The method of claim 1 further comprisingresynchronizing, by the storage management computing device, data andconfiguration planes to the healed at least one primary node.
 6. Themethod of claim 1 further comprising receiving, by the storagemanagement computing device, a command to switch from the use of the atleast one primary node to the use of the at least one secondary storagenode.
 7. The method of claim 6 further comprising automatically, by thestorage management computing device, determining and implementingswitchover tasks within processing resources of the storage network. 8.The method of claim 7 further comprising managing which nodes are to beswitched in response to a switchover command.
 9. A storage managementcomputing device comprising: a processor; a memory coupled to theprocessor which is configured to be capable of executing programmedinstructions comprising and stored in the memory to: replicate a dataplane and a configuration plane of at least one primary node to at leastone secondary storage node for a client device in communication with astorage network; monitor the at least one primary node for a failure incommunication within the storage network; switch primary access for theclient device from the at least one primary node to the at least onesecondary storage node in the event that a failure in the at least oneprimary node is recognized, the switching further comprising healing theat least one failed primary node and switching primary access for theclient device back from the at least one secondary storage node to thehealed at least one primary node; and provide a switching statusnotification indicating progress of the switching primary access for theclient device from the at least one primary node to the at least onesecondary storage node and a healing status notification indicatingprogress of healing the at least one failed primary node.
 10. The deviceof claim 9 wherein replicating a data plane comprises replicating anasynchronous resource and a synchronous resource configured to storedata from the client device.
 11. The device of claim 9 whereinreplicating a configuration plane comprises replicating a virtualserver.
 12. The device of claim 11 wherein the replicated virtual serveris configured to be dormant on the at least one secondary storage nodewhile the at least one primary node is functioning.
 13. The device ofclaim 9 wherein the processor coupled to the memory is furtherconfigured to be capable of executing at least one additional programmedinstruction comprising and stored in the memory to resynchronize dataand configuration planes to the healed at least one primary node. 14.The device of claim 9 wherein the processor coupled to the memory isfurther configured to be capable of executing at least one additionalprogrammed instruction comprising and stored in the memory to receive acommand to switch from the use of the at least one primary node to theuse of the at least one secondary storage node.
 15. The device of claim14 wherein the processor coupled to the memory is further configured tobe capable of executing at least one additional programmed instructioncomprising and stored in the memory to automatically determine andimplementing switchover tasks within processing resources of the storagenetwork.
 16. The device of claim 15 wherein the processor coupled to thememory is further configured to be capable of executing at least oneadditional programmed instruction comprising and stored in the memory tomanage which nodes are to be switched in response to a switchovercommand.
 17. A non-transitory computer readable medium having storedthereon instructions for administering disaster recovery resources in astorage network comprising executable code which when executed by aprocessor, causes the processor to perform steps comprising: replicatinga data plane and a configuration plane of at least one primary node toat least one secondary storage node; monitoring the at least one primarynode for a failure in communication within the storage network;switching primary access for the client device from the at least oneprimary node to the at least one secondary storage node in the eventthat a failure in the at least one primary node is recognized, theswitching further comprising healing the at least one failed primarynode and switching primary access for the client device back from the atleast one secondary storage node to the healed at least one primarynode; and providing a switching status notification indicating progressof the switching primary access for the client device from the at leastone primary node to the at least one secondary storage node and ahealing status notification indicating progress of healing the at leastone failed primary node.
 18. The medium claim 17 wherein replicating aconfiguration plane comprises replicating a virtual server.
 19. Themedium of claim 18 wherein the replicated virtual server is configuredto be dormant on the at least one secondary storage node while the atleast one primary node is functioning.
 20. The medium of claim 17further comprising resynchronizing data and configuration planes to thehealed at least one primary node.
 21. The medium of claim 17 furthercomprising receiving a command to switch from the use of the at leastone primary node to the use of the at least one secondary storage node.22. The medium of claim 21 further comprising automatically determiningand implementing switchover tasks within processing resources of thestorage network.
 23. The medium of claim 22 further comprising managingwhich nodes are to be switched in response to a switchover command.